JupyterHub HPC Meeting - November#
Date: 2020-11-04
Time: 8:30 AM PST
GitHub issue:
Calendar for future meetings: https://jupyterhub-team-compass.readthedocs.io/en/latest/meetings.html
Welcome to the Meeting#
Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:
name / institution / GitHub handle
Rollin / NERSC / @rcthomas
Shreyas Cholia / NERSC|LBL / @shreddd
Zach Price / ORNL
Mike Milligan / MSI@UMN / @mbmilligan
Kevin Paul / NCAR / @kmpaul
Félix-Antoine Fortin / Compute Canada - Université Laval / @cmd-ntrf
Jeffrey Miller / ORNL / @millerjl1701
Ryan Prout / ORNL
Spencer Ward / ORNL
Quick updates#
60 second updates on things you have been up to, questions you have, or developments you think people should know about. This is also a chance to suggest a future presentation if you’ve got work currently in progress you might want to share. Please add yourself, and if you do not have an update to share, you can pass.
Rollin: Working on Shifter (Docker) image selection at Hub service, prototype working, could demo at a future meeting, targeting Dec for deployment
Rollin: Moving deployment from a Rancher 1 to Rancher 2 (k8s) deployment also targeting Dec for deployment, could talk about that also at future meeting
Mike: MSI is moving main cluster from Torque to SLURM, so work ongoing to convert and update our deployment
Need help from Carol to set up pushing to PyPI
Dummy release of batchspawner 1.0.1, using GH actions
Work in progress (see below)
Kevin: We have a release of Jupyter-Forward which is serving as our alternative for not have sysadmin privileges on our JupyterHub. Curious to know what other HPCers think about tools like this.
Reports and celebrations#
This is a place to make announcements (without a need for discussion). This is also a great place to give shout-outs to contributors! We’ll read through these at the beginning of the meeting.
New release of Jupyter-Forward
Command line tool from remote machine
Bundles up setting up ssh tunnel etc
Side-steps hub
Launch command can be submission to queue
Puts the responsibility of intentionally waiting for a resource with a lot of contention more clearly onto the user (instead of waiting at hub all day)
What about expanding scope beyond JupyterLab (start other things with it) there are other tools for which there is no “hub” – Félix
add item here
add item here
Agenda items#
Let’s collect all potential agenda items here before the start of the meeting. We will then attempt to create a coherent agenda that fits in the 60m meeting slot. If there are similar items try and group them together.
Shreyas, Michael, Rollin, et al (10-15 min?): JupyterCon retrospective
Enterprise deployments, a few HPC deployment talks, mostly known to us already
Notebooker etc might contain a lot of useful or reusable patterns, things that might be worth deploying for users
Xeus kernel, debuggers, but not a plan to mainline those into the standard kernels
Zach (20-30 min?): JupyterHub setups at ORNL
JupyterHub is a “birthright”
At ORNL JupyterHub gets deployed in various places upon request from researchers
Pushing the idea of a central entrypoint for JupyterHub, use BatchSpawner, WrapSpawner, ProfileSpawner to see what resources there are
Otherwise it’s repeating the same process every time with every group from the ground up
Need higher availability for a institution-wide solution, would probably start with a lower user count but would go up like deploying gitlab etc (but is the hub the HA problem actually, it isn’t at at NERSC really…)
Multiple security zones, LDAP instances that can’t be used, makes things more challenging
“ProfileSpawner” for Authenticators? Is that a JupyterHub responsibility or some other solution (Dex oauth)? JWT?
JupyterLab 3 + JupyterHub - does anyone know if the integration works? - Shreyas
Rollin says no (haven’t tried to run off master but that may make it work)
Mike - trying to debug Github Actions pipeline for pushing batchspawner/wrapspawner releases to PyPI
Fixed a bug during the meeting
Rollin - Fake batch spawner in BatchSpawner so I don’t have to handle sshspawner separately, which needs a bunch of the same logic?
What about systemd (Zach)?
add item here (include your name and estimated time for discussion).