Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set sticky bit on connection files #201

Merged
merged 3 commits into from
Feb 7, 2017

Conversation

minrk
Copy link
Member

@minrk minrk commented Sep 13, 2016

avoids periodic cleanup of runtime directory

closes #199

@minrk
Copy link
Member Author

minrk commented Sep 13, 2016

Alternative: we could set the sticky bit on the runtime_dir itself.

@takluyver
Copy link
Member

The spec only mentions setting the sticky bit on files, and whatever does the cleanup is probably running as root, so it wouldn't be bound by the sticky bit. So let's stick with doing it on the file rather than the directory.

@@ -135,6 +136,21 @@ def write_connection_file(fname=None, shell_port=0, iopub_port=0, stdin_port=0,
with open(fname, 'w') as f:
f.write(json.dumps(cfg, indent=2))

if hasattr(stat, 'S_ISVTX'):
# set the sticky bit to avoid periodic cleanup
permissions = os.stat(fname).st_mod
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

st_mode

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, thanks

@minrk
Copy link
Member Author

minrk commented Sep 13, 2016

The error in #199 suggests that the runtime_dir was cleaned up because writing a new file should only fail if the directory doesn't exist.

avoids periodic cleanup of runtime directory
@minrk
Copy link
Member Author

minrk commented Sep 13, 2016

Now that I think about it, since #199 suggests that the directory is what was cleaned up, we may need something directory-related. Either ensure the directory exists when we try to write a connection file, or set the sticky bit on our runtime_dir.

@minrk
Copy link
Member Author

minrk commented Sep 13, 2016

Or do something non-sticky like call os.utime every once in a while to indicate that a given runtime file is still 'active'.

@takluyver
Copy link
Member

Ah, right, we make a subdirectory inside the runtime dir. I'd forgotten that. Yes, in that case we probably need this sticky bit on the jupyter directory.

in addition to connection file

to avoid cleanup of empty dir during long-idle servers
connection file is in CWD, no need for sticky bits
@minrk
Copy link
Member Author

minrk commented Feb 7, 2017

Finally got back to this one. It now sets the sticky bit on the directory containing the connection file, as well.

@minrk minrk added this to the 5.0 milestone Feb 7, 2017
@takluyver takluyver merged commit e3f54d0 into jupyter:master Feb 7, 2017
@AlJohri
Copy link

AlJohri commented Jul 10, 2017

@minrk it seems like setting the sticky bit consistently fails on macOS because of a permission issue with the /var directory:

/Users/johria/.virtualenvs/heliograf-analysis-hss/lib/python3.6/site-packages/jupyter_client/connect.py:157: RuntimeWarning: Failed to set sticky bit on '/var/folders/c5/sxpknfp571v3ydglf4305g9m0000gn/T': [Errno 1] Operation not permitted: '/var/folders/c5/sxpknfp571v3ydglf4305g9m0000gn/T'
  RuntimeWarning,

@minrk minrk deleted the sticky-connection-file branch July 11, 2017 07:41
@BoPeng
Copy link
Contributor

BoPeng commented Aug 8, 2017

I am seeing the same error on Mac.

/Users/bpeng1/anaconda/lib/python3.6/site-packages/jupyter_client/connect.py:157: 
RuntimeWarning: Failed to set sticky bit on '/var/folders/ys/gnzk0qbx5wbdgm531v82xxljv5yqy8/T': [Errno 1] 
Operation not permitted: '/var/folders/ys/gnzk0qbx5wbdgm531v82xxljv5yqy8/T'

@takluyver
Copy link
Member

We're already catching the error and issuing a warning, which is what you see. Maybe we should skip the warning on Mac since it probably doesn't matter.

@BoPeng
Copy link
Contributor

BoPeng commented Aug 8, 2017

I frankly do not know if it matters but everything appears to be ok after the warning. The warning is pretty low level and scary so it is perhaps better to remove it (at least on mac).

@takluyver
Copy link
Member

It shouldn't matter - setting the sticky bit is part of an XDG specification (XDG_RUNTIME_DIR), which as far as I know is only used on free desktop systems (Linux, BSD, etc.).

@AlJohri
Copy link

AlJohri commented Aug 8, 2017

I run a command to rexecute a jupyter notebook in my Makefile and the output warning looks a bit scary for those unfamiliar. I'd give a +1 for either not setting the sticky bit on mac or suppressing the warning on macos.

@minrk
Copy link
Member Author

minrk commented Aug 9, 2017

@AlJohri Can you show the ls -la permissions on the directory? Are you its owner?

I'm okay skipping this on mac, but it's a bit surprising because I don't get an permission errors doing this, so I'd like to understand it, as well.

@AlJohri
Copy link

AlJohri commented Aug 9, 2017

sure, I think /var is traditionally owned by root on macOS? I think I've run into this while trying to put some pid files in var run before. This particular path may be different though.

Als-MacBook-Pro # pwd
/var
Als-MacBook-Pro # ls -la
total 0
drwxr-xr-x  25 root          wheel          850 Jul 24 02:01 ./
drwxr-xr-x@  6 root          wheel          204 May 28 09:13 ../
drwx------   2 root          wheel           68 Jul 30  2016 agentx/
drwxr-xr-x   8 daemon        wheel          272 Sep 13  2016 at/
drwx------  25 root          wheel          850 Aug  8 11:15 audit/
drwx------   2 root          wheel           68 Jul 30  2016 backups/
drwxr-xr-x  70 root          wheel         2380 Aug  8 17:45 db/
drwxr-xr-x   2 root          sys             68 Jul 30  2016 empty/
drwxr-xr-x   4 root          wheel          136 Jul 20 11:28 folders/
drwxr-x---   2 _jabber       _jabber         68 Jul 30  2016 jabberd/
drwxr-xr-x   4 root          wheel          136 May 28 11:22 lib/
drwxr-xr-x  61 root          wheel         2074 Aug  9 08:29 log/
drwxr-x---   2 _mobileasset  _mobileasset    68 Jul 30  2016 ma/
drwxrwxr-x   2 root          mail            68 Jul 30  2016 mail/
drwxr-xr-x   3 root          wheel          102 Jul 30  2016 msgs/
drwxr-xr-x   2 root          wheel           68 Jul 30  2016 netboot/
drwxr-xr-x   5 _networkd     _networkd      170 May 28 09:16 networkd/
drwxr-x---   6 root          wheel          204 May 28 10:39 root/
drwxr-xr-x   4 root          wheel          136 Jul 30  2016 rpc/
drwxrwxr-x  31 root          daemon        1054 Aug  9 08:27 run/
drwxr-xr-x   2 daemon        wheel           68 Jul 30  2016 rwho/
drwxr-xr-x   6 root          wheel          204 Sep 13  2016 spool/
drwxrwxrwt   6 root          wheel          204 Aug  9 08:33 tmp/
drwxr-xr-x   4 root          wheel          136 Aug  8 12:22 vm/
drwxr-xr-x   3 root          wheel          102 Jul 30  2016 yp/
Als-MacBook-Pro # cd /var/folders/
Als-MacBook-Pro # ls -la
total 0
drwxr-xr-x   4 root  wheel  136 Jul 20 11:28 ./
drwxr-xr-x  25 root  wheel  850 Jul 24 02:01 ../
drwxr-xr-x@  3 root  wheel  102 May 28 09:27 c5/
drwxr-xr-x@ 17 root  wheel  578 May 28 20:17 zz/
Als-MacBook-Pro # pwd
/var/folders/c5/sxpknfp571v3ydglf4305g9m0000gn/T
Als-MacBook-Pro # ls -lha
total 205304
drwx------@ 78 johria  staff   2.6K Aug  9 08:33 ./
drwxr-xr-x@  6 johria  staff   204B May 28 10:24 ../
drwx------   7 johria  staff   238B Aug  8 12:02 .AddressBookLocks/
drwx------   4 johria  staff   136B Aug  8 11:16 .BQR82RBBHL.com.tinyspeck.slackmacgap.NISTG4/
drwx------   2 johria  staff    68B Aug  8 11:15 .CalendarLocks/
drwx------@  4 johria  staff   136B Aug  8 11:15 .com.google.Chrome.dumnj0/
drwx------@  4 johria  staff   136B Aug  8 11:15 .io.nwjs.nw.KBzpRZ/
drwx------   2 johria  staff    68B Aug  8 11:15 .io.nwjs.nw.nFadVj/
-rw-r--r--   1 johria  staff     0B Aug  8 11:15 .keystoneAgentLock
drwx------@  2 johria  staff    68B Aug  8 11:15 2BUA8C4S2C.com.agilebits.onepassword-osx-helper/
drwxr-xr-x   2 johria  staff    68B Aug  8 13:24 MSau_10168/
-rw-------   1 johria  staff   559B Aug  8 11:59 SOSBackup-PCS-MasterKey-tomb
drwx------   3 johria  staff   102B Aug  8 23:53 SetupClient/
drwx------   2 johria  staff    68B Aug  9 08:33 TemporaryItems/
drwx------   3 johria  staff   102B Aug  8 11:15 assistantd/
drwx------@  3 johria  staff   102B Aug  8 11:15 chrome-1L4kxf/
drwx------@  3 johria  staff   102B Aug  8 11:15 chrome-J3KBSh/
drwx------@  2 johria  staff    68B Aug  8 11:47 com.agilebits.onepassword-osx.safariextensioncompanion/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.agilebits.onepasswordnativemessaginghost/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.AddressBook.ContactsAccountsService/
drwx------@  2 johria  staff    68B Aug  8 20:07 com.apple.AddressBook.FaceTimeService/
drwxr-xr-x   2 johria  staff    68B Aug  8 20:07 com.apple.AddressBook.thumbnailcache/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.AirPlayUIAgent/
drwx------@  3 johria  staff   102B Aug  8 11:15 com.apple.CalendarAgent/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.CalendarNotification.CalNCService/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.CloudPhotosConfiguration/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.ContactsAgent/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.MailCacheDelete/
drwx------@  4 johria  staff   136B Aug  8 18:16 com.apple.Notes/
drwx------@  2 johria  staff    68B Aug  8 11:17 com.apple.OSDUIHelper/
drwx------@  2 johria  staff    68B Aug  8 11:16 com.apple.PhotoIngestService/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.PressAndHold/
drwxr-xr-x@  3 johria  staff   102B Aug  8 11:47 com.apple.Safari/
drwx------@  3 johria  staff   102B Aug  8 11:15 com.apple.Safari.CacheDeleteExtension/
drwx------   2 johria  staff    68B Aug  8 11:16 com.apple.Safari.History/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.Siri/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.SocialPushAgent/
drwx------@  3 johria  staff   102B Aug  8 13:45 com.apple.WeatherKitService/
drwx------   2 johria  staff    68B Aug  8 11:15 com.apple.bird/
drwx------@  3 johria  staff   102B Aug  8 11:16 com.apple.cloudphotosd/
drwx------@  2 johria  staff    68B Aug  8 11:47 com.apple.corerecents.recentsd/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.ctkahp/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.dt.IDECacheDeleteAppExtension/
drwxr-xr-x   3 johria  staff   102B Aug  8 17:25 com.apple.finder/
drwx------@  3 johria  staff   102B Aug  8 11:47 com.apple.geod/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.iBooksX.CacheDelete/
drwx------@  2 johria  staff    68B Aug  8 16:06 com.apple.iCal/
drwx------@  2 johria  staff    68B Aug  8 13:45 com.apple.iCal.CalendarNC/
drwx------@  3 johria  staff   102B Aug  8 11:17 com.apple.iChat/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.iTunesCacheExtension/
drwx------@  2 johria  staff    68B Aug  8 17:46 com.apple.lateragent/
drwx------@  2 johria  staff    68B Aug  8 17:46 com.apple.mediaanalysisd/
drwx------@  2 johria  staff    68B Aug  8 13:45 com.apple.ncplugin.stocks/
drwx------@  2 johria  staff    68B Aug  8 13:45 com.apple.ncplugin.weather/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.notificationcenterui.WeatherSummary/
drwx------@  3 johria  staff   102B Aug  8 17:45 com.apple.photoanalysisd/
drwx------@  3 johria  staff   102B Aug  8 11:16 com.apple.photolibraryd/
drwx------@  3 johria  staff   102B Aug  8 11:16 com.apple.photomoments/
drwx------@  2 johria  staff    68B Aug  8 17:24 com.apple.quicklook.ui.helper/
drwx------@  2 johria  staff    68B Aug  8 17:44 com.apple.siri.media-indexer/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.apple.soagent/
drwx------   2 johria  staff    68B Aug  8 11:15 com.apple.tccd/
drwx------@  2 johria  staff    68B Aug  8 13:16 com.apple.tonelibraryd/
drwx------@  2 johria  staff    68B Aug  8 11:15 com.docker.helper/
-rw-------   1 johria  staff   377K Aug  8 12:01 com.dropbox.DropboxMacUpdate.33.3.14.dmg
-rw-------   1 johria  staff   100M Aug  8 12:01 com.getdropbox.dropbox.33.3.14.dmg
drwx------@  2 johria  staff    68B Aug  8 11:15 com.getdropbox.dropbox.garcon/
drwx------@  3 johria  staff   102B Aug  8 13:24 com.microsoft.Office365ServiceV2/
drwx------@  5 johria  staff   170B Aug  8 13:24 com.microsoft.Outlook/
drwxr-xr-x   7 johria  staff   238B Aug  8 11:15 hsperfdata_johria/
drwxr-xr-x   2 johria  staff    68B Aug  8 11:15 jetty-0.0.0.0-8983-webapp-_solr-any-7515977943968857408.dir/
drwxr-xr-x   2 johria  staff    68B Aug  8 11:16 jetty-127.0.0.1-7474-browser-_browser-any-6981420697006606178.dir/
drwxr-xr-x   2 johria  staff    68B Aug  8 11:15 jna--1154555289/
drwxr-xr-x   3 johria  staff   102B Aug  8 20:09 shared-pasteboard/
drwxr-xr-x   4 johria  staff   136B Aug  9 08:30 sp_update/
drwx------   2 johria  staff    68B Aug  8 12:27 ssh-VRtvsp23Sc5t/
-rw-------   1 johria  staff   165B Aug  8 11:15 start_1172299703324100109.properties
-rw-------   1 johria  staff   536B Aug  8 11:15 xcrun_db

@minrk
Copy link
Member Author

minrk commented Aug 9, 2017

Strange. That looks like you do have permission to set permissions on /var/folders/c5/sxpknfp571v3ydglf4305g9m0000gn/T, which is what's failing above. Can you run:

import os
import stat
path = '/var/folders/c5/sxpknfp571v3ydglf4305g9m0000gn/T'
permissions = os.stat(path).st_mode
new_permissions = permissions | stat.S_ISVTX
os.chmod(path, new_permissions)

@AlJohri
Copy link

AlJohri commented Aug 9, 2017

acc-16249 # python
Python 3.6.2 (default, Jul 25 2017, 11:20:33)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import stat
>>> path = '/var/folders/c5/sxpknfp571v3ydglf4305g9m0000gn/T'
>>> permissions = os.stat(path).st_mode
>>> new_permissions = permissions | stat.S_ISVTX
>>> os.chmod(path, new_permissions)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [Errno 1] Operation not permitted: '/var/folders/c5/sxpknfp571v3ydglf4305g9m0000gn/T'

@minrk
Copy link
Member Author

minrk commented Aug 9, 2017

And what about:

stat /var/folders/c5/sxpknfp571v3ydglf4305g9m0000gn /var/folders/c5/sxpknfp571v3ydglf4305g9m0000gn/T

?

@AlJohri
Copy link

AlJohri commented Aug 9, 2017

$ stat /var/folders/c5/sxpknfp571v3ydglf4305g9m0000gn /var/folders/c5/sxpknfp571v3ydglf4305g9m0000gn/T
16777220 437605 drwxr-xr-x 6 johria staff 0 204 "Aug  9 08:35:15 2017" "May 28 10:24:36 2017" "May 28 10:24:36 2017" "May 28 09:27:33 2017" 4096 0 0x100000 /var/folders/c5/sxpknfp571v3ydglf4305g9m0000gn
16777220 437608 drwx------ 78 johria staff 0 2652 "Aug  9 08:35:33 2017" "Aug  9 09:49:33 2017" "Aug  9 09:49:33 2017" "May 28 09:27:33 2017" 4096 0 0x100000 /var/folders/c5/sxpknfp571v3ydglf4305g9m0000gn/T

@minrk
Copy link
Member Author

minrk commented Aug 9, 2017

Weird! All evidence points to you having permission to set the bit, I don't know why it wouldn't work. In any case, #286 should suppress the warning in this particular case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

write_connection_file fails in long running notebook server
5 participants