JupyterHub HPC Meeting - November#

Welcome to the Meeting#

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

  • name / institution / GitHub handle

  • Rollin / NERSC / @rcthomas

  • Shreyas Cholia / NERSC|LBL / @shreddd

  • Zach Price / ORNL

  • Mike Milligan / MSI@UMN / @mbmilligan

  • Kevin Paul / NCAR / @kmpaul

  • Félix-Antoine Fortin / Compute Canada - Université Laval / @cmd-ntrf

  • Jeffrey Miller / ORNL / @millerjl1701

  • Ryan Prout / ORNL

  • Spencer Ward / ORNL

Quick updates#

60 second updates on things you have been up to, questions you have, or developments you think people should know about. This is also a chance to suggest a future presentation if you’ve got work currently in progress you might want to share. Please add yourself, and if you do not have an update to share, you can pass.

  • Rollin: Working on Shifter (Docker) image selection at Hub service, prototype working, could demo at a future meeting, targeting Dec for deployment

  • Rollin: Moving deployment from a Rancher 1 to Rancher 2 (k8s) deployment also targeting Dec for deployment, could talk about that also at future meeting

  • Mike: MSI is moving main cluster from Torque to SLURM, so work ongoing to convert and update our deployment

    • Need help from Carol to set up pushing to PyPI

    • Dummy release of batchspawner 1.0.1, using GH actions

    • Work in progress (see below)

  • Kevin: We have a release of Jupyter-Forward which is serving as our alternative for not have sysadmin privileges on our JupyterHub. Curious to know what other HPCers think about tools like this.

Reports and celebrations#

This is a place to make announcements (without a need for discussion). This is also a great place to give shout-outs to contributors! We’ll read through these at the beginning of the meeting.

  • New release of Jupyter-Forward

    • Command line tool from remote machine

    • Bundles up setting up ssh tunnel etc

    • Side-steps hub

    • Launch command can be submission to queue

    • Puts the responsibility of intentionally waiting for a resource with a lot of contention more clearly onto the user (instead of waiting at hub all day)

    • What about expanding scope beyond JupyterLab (start other things with it) there are other tools for which there is no “hub” – Félix

  • add item here

  • add item here

Agenda items#

Let’s collect all potential agenda items here before the start of the meeting. We will then attempt to create a coherent agenda that fits in the 60m meeting slot. If there are similar items try and group them together.

  • Shreyas, Michael, Rollin, et al (10-15 min?): JupyterCon retrospective

    • Enterprise deployments, a few HPC deployment talks, mostly known to us already

    • Notebooker etc might contain a lot of useful or reusable patterns, things that might be worth deploying for users

    • Xeus kernel, debuggers, but not a plan to mainline those into the standard kernels

  • Zach (20-30 min?): JupyterHub setups at ORNL

    • JupyterHub is a “birthright”

    • At ORNL JupyterHub gets deployed in various places upon request from researchers

    • Pushing the idea of a central entrypoint for JupyterHub, use BatchSpawner, WrapSpawner, ProfileSpawner to see what resources there are

    • Otherwise it’s repeating the same process every time with every group from the ground up

    • Need higher availability for a institution-wide solution, would probably start with a lower user count but would go up like deploying gitlab etc (but is the hub the HA problem actually, it isn’t at at NERSC really…)

    • Multiple security zones, LDAP instances that can’t be used, makes things more challenging

    • “ProfileSpawner” for Authenticators? Is that a JupyterHub responsibility or some other solution (Dex oauth)? JWT?

  • JupyterLab 3 + JupyterHub - does anyone know if the integration works? - Shreyas

    • Rollin says no (haven’t tried to run off master but that may make it work)

  • Mike - trying to debug Github Actions pipeline for pushing batchspawner/wrapspawner releases to PyPI

    • Fixed a bug during the meeting

  • Rollin - Fake batch spawner in BatchSpawner so I don’t have to handle sshspawner separately, which needs a bunch of the same logic?

    • What about systemd (Zach)?

  • add item here (include your name and estimated time for discussion).