JupyterHub HPC Meeting - May 2021#

Welcome to the Meeting#

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

  • name / institution / GitHub handle

  • Rollin / NERSC / @rcthomas

  • Shane / NERSC / @scanon

  • Zach Price / ORNL

  • Richard / Aalto / rkdarst

  • Aric Werner / NCAR / @aawerner

  • Jared Baker / NCAR / @jbaksta

  • Rick Wagner / UCSD / @rpwagner

  • Félix-Antoine Fortin / Université Laval /@cmd-ntrf

  • Mary Thomas / UCSD, SDSC / @marypthomas

  • Scott Sakai / UCSD, SDSC / @scottsakai

  • Jens Henrik Goebbert / FZJ, JSC / @jhgoebbert

  • Thomas Mendoza / LLNL / @tgmachina

Quick updates#

60 second updates on things you have been up to, questions you have, or developments you think people should know about. This is also a chance to suggest a future presentation if you’ve got work currently in progress you might want to share. Please add yourself, and if you do not have an update to share, you can pass.

  • NAME: What you’d like to update on

  • Rick, Rollin Jupyter Community Workshop series is coming back to life, “plans for plans” for security-focused workshop will be coming together soon.

    • Want to announce dates and place in Sep

    • Need to make sure people can travel, be in same place when we have workshop

    • If you have ideas to share on security please feel free to let us know about it by email

Reports and celebrations#

This is a place to make announcements (without a need for discussion). This is also a great place to give shout-outs to contributors! We’ll read through these at the beginning of the meeting.

  • Wrapspawner release 1.0

  • Michael is out this month on paternity leave

Agenda items#

Let’s collect all potential agenda items here before the start of the meeting. We will then attempt to create a coherent agenda that fits in the 60m meeting slot. If there are similar items try and group them together.

  • SDSC Jupyter Proxy Service

    • Project is called “Satellite”

    • Jupyter initially appeared as “plain-text back door into the cluster”

    • “Make the right things easier than the wrong things”

    • Satellite:

      • Invoke client, get URL, notebook shows up

      • Self-service https reverse-proxy

      • Shell-based utility to orchestrate RP and Slurm

      • sdsc-hpc-training-org/satellite

    • Reverse proxy:

    • start-jupyter, Galyleo:

      • Batch jobs, issues with templates and integrating Singularity

      • Open OnDemand portal on Expanse (have Matlab, RStudio, want to integrate Satellite)

      • Galyleo repo: sdsc-hpc-training-org/satellite

    • –transport=ipc

    • Streetwise guide: https://docs.google.com/presentation/d/1X_0hHYpQcCwrwbPMGSHuTeys7KPW3bC1/edit#slide=id.g8cad2b19f4_0_1024

    • Shane is trying it on expanse during the call :)

    • Questions:

      • What proxy service are you using (traefik, CHP?)

        • Looks like it is dynamically configuring apache

      • Is their an open repo for the API? (Answered)

      • Is there any user documentation? I couldn’t quickly find it via Google. (Answered)

      • Rather obvious one: Hub enables subdomains, provides proxy; what were the issues w/Hub that made you think you needed your own approach?

        • Already have trust established (Q: e2e SSL)

        • Issues local to SDSC when starting w/Hub

        • Want to prevent users from running Jupyter themselves full stop (if they go around Hub)

        • Concern about where hub lives with respect to trust boundary of cluster, can be addresses

      • Slides available? distributable?

        • yes, and yes

  • Standing Items:

    • Batchspawner check-in

      • Need to check in with outstanding PRs now that autoformat PR is merged, will need to rebase

    • Wrapspawner check-in

      • At last call, goal was release, then deep dive on issues

      • What are the top issues?

    • JupyterLab 3 deployment news

      • NERSC

        • Pushed back to May 19 at NERSC but will happen

        • ~All needed extensions are working now

        • One issue found and fixed in jupyter_server