JupyterHub HPC Meeting - November

• Date: 2020-11-04

• Time: 8:30 AM PST

  • Your timezone: https://arewemeetingyet.com/Los Angeles/2020-11-04/08:30/JupyterHub-HPC

• GitHub issue:

• Calendar for future meetings: https://jupyterhub-team-compass.readthedocs.io/en/latest/meetings.html

Welcome to the Meeting

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

• name / institution / GitHub handle

• Rollin / NERSC / @rcthomas

• Shreyas Cholia / NERSC|LBL / @shreddd

• Zach Price / ORNL

• Mike Milligan / MSI@UMN / @mbmilligan

• Kevin Paul / NCAR / @kmpaul

• Félix-Antoine Fortin / Compute Canada - Université Laval / @cmd-ntrf

• Jeffrey Miller / ORNL / @millerjl1701

• Ryan Prout / ORNL

• Spencer Ward / ORNL

Quick updates

60 second updates on things you have been up to, questions you have, or developments you think people should know about. This is also a chance to suggest a future presentation if you’ve got work currently in progress you might want to share. Please add yourself, and if you do not have an update to share, you can pass.

• Rollin: Working on Shifter (Docker) image selection at Hub service, prototype working, could demo at a future meeting, targeting Dec for deployment

• Rollin: Moving deployment from a Rancher 1 to Rancher 2 (k8s) deployment also targeting Dec for deployment, could talk about that also at future meeting

• Mike: MSI is moving main cluster from Torque to SLURM, so work ongoing to convert and update our deployment

  • Need help from Carol to set up pushing to PyPI

  • Dummy release of batchspawner 1.0.1, using GH actions

  • Work in progress (see below)

• Kevin: We have a release of Jupyter-Forward which is serving as our alternative for not have sysadmin privileges on our JupyterHub. Curious to know what other HPCers think about tools like this.

Reports and celebrations

This is a place to make announcements (without a need for discussion). This is also a great place to give shout-outs to contributors! We’ll read through these at the beginning of the meeting.

• New release of Jupyter-Forward

  • Command line tool from remote machine

  • Bundles up setting up ssh tunnel etc

  • Side-steps hub

  • Launch command can be submission to queue

  • Puts the responsibility of intentionally waiting for a resource with a lot of contention more clearly onto the user (instead of waiting at hub all day)

  • What about expanding scope beyond JupyterLab (start other things with it) there are other tools for which there is no “hub” – Félix

• add item here

• add item here

Agenda items

Let’s collect all potential agenda items here before the start of the meeting. We will then attempt to create a coherent agenda that fits in the 60m meeting slot. If there are similar items try and group them together.

• Shreyas, Michael, Rollin, et al (10-15 min?): JupyterCon retrospective

  • Enterprise deployments, a few HPC deployment talks, mostly known to us already

  • Notebooker etc might contain a lot of useful or reusable patterns, things that might be worth deploying for users

  • Xeus kernel, debuggers, but not a plan to mainline those into the standard kernels

• Zach (20-30 min?): JupyterHub setups at ORNL

  • JupyterHub is a “birthright”

  • At ORNL JupyterHub gets deployed in various places upon request from researchers

  • Pushing the idea of a central entrypoint for JupyterHub, use BatchSpawner, WrapSpawner, ProfileSpawner to see what resources there are

  • Otherwise it’s repeating the same process every time with every group from the ground up

  • Need higher availability for a institution-wide solution, would probably start with a lower user count but would go up like deploying gitlab etc (but is the hub the HA problem actually, it isn’t at at NERSC really…)

  • Multiple security zones, LDAP instances that can’t be used, makes things more challenging

  • “ProfileSpawner” for Authenticators? Is that a JupyterHub responsibility or some other solution (Dex oauth)? JWT?

• JupyterLab 3 + JupyterHub - does anyone know if the integration works? - Shreyas

  • Rollin says no (haven’t tried to run off master but that may make it work)

• Mike - trying to debug Github Actions pipeline for pushing batchspawner/wrapspawner releases to PyPI

  • Fixed a bug during the meeting

• Rollin - Fake batch spawner in BatchSpawner so I don’t have to handle sshspawner separately, which needs a bunch of the same logic?

  • What about systemd (Zach)?

• add item here (include your name and estimated time for discussion).