Ctrl+K

JupyterHub HPC Meeting - November 2021#

Welcome to the Meeting#

Hello! If you are joining the team video meeting, sign in below so we know who was here. Roll call:

  • name / institution / GitHub handle

  • Min RK / Simula / @minrk

  • Jens Henrik Göbbert / Jülich Supercomputing Centre / @j.goebbert

  • Zach Price / ORNL /

  • Shreyas Cholia / LBL / @shreddd

  • Richard Darst / Aalto / @rkdarst

  • Michael Milligan / MSI @ UMN / @mbmilligan

  • Félix-Antoine Fortin / Compute Canada - Université Laval / @cmd-ntrf

  • Simon Li / University of Dundee / @manics

Quick updates#

60 second updates on things you have been up to, questions you have, or developments you think people should know about. This is also a chance to suggest a future presentation if you’ve got work currently in progress you might want to share. Please add yourself, and if you do not have an update to share, you can pass.

  • Name: Your update

Reports and celebrations#

This is a place to make announcements (without a need for discussion). This is also a great place to give shout-outs to contributors! We’ll read through these at the beginning of the meeting.

  • Name: Your report or celebration

Agenda items#

Let’s collect all potential agenda items here before the start of the meeting. We will then attempt to create a coherent agenda that fits in the 60m meeting slot. If there are similar items try and group them together.

  • @j.goebbert redesign of JupyterHub landing page

    • More and more features on the JSC landing/home page

    • Users were getting confused, getting passed from one page to another

    • The options spawner-type functionality comes up as a modal on the home page

    • For every system, for every JupyterLab, for login nodes, for every kind of compute node

    • Each user starts with a short list of defaults and then they can come up with more options

    • What’s available depends on what’s been installed on the file system

    • Spawner starts a script that decides what to do

    • Ability to let users add their own JupyterLab with their own weird extensions

    • Can a user share a configuration with with other users?

      • The hub picks up the installation so users can share with each other this way

      • Installation on a project’s space on the file system shares it with everone else on the project

    • Is there a standard “base” set of extensions that power users can use to create their own?

      • Based on easy build so users can base off that

      • Curating some extensions (taking away…)

      • Lots of extensions are a burden…

    • Modules used to extend the labextension search path

      • Done dynamically in the start script

      • Not part of the install/configuration

  • Simon BinderHub in a Box

    • Runs on k8s, scales well, really powerful, but it is a pain to install locally for testing

    • Try to host BinderHub just on Docker? Turns out not as difficult as you might think

    • Started a bigger discussion about abstraction layers?

    • Need: r2d, hub (nothing that says k8s is intrinsic)

    • Main PR: jupyterhub/binderhub#1364

    • Demo: https://katacoda.com/manics/scenarios/binderhub-in-a-box

      • Platform for infrastructure learning

      • Spins up a VM… actually look at the demo, it’s very nicely documented

      • Repo2docker runs locally, so for more than a few users it needs to be pretty beefy VM

      • Demo is done with docker, but you can use podman (podman daemon spawner, and fake docker registry)

      • BinderHub is running as a JupyterHub service here

    • Minimizing Privilege for Building HPC Containers: https://arxiv.org/pdf/2104.07508.pdf

  • k3s some exploration, it can be done, but teaching k8s is tedious :)

  • SC21 activities?

    • No Interactive HPC workshop

    • Felix is demo’ing latest Magic Castle at Training and Tools HPC workshop

  • ISC22

    • Program committee?

      • Shreyas find people will do everything from his group

  • JupyterHub 2.0

    • PEOPLE SHOULD TEST

    • RBAC

      • Roles and scopes, how permissions work in JupyterHub

      • More granular permissions: useful for collaboration and RTC in JupyterLab

    • Spawning changes

      • Small change, JupyterHub will not pass its own args as command line -> environment variables

      • The only ones passed are the ones that user configuration has explicitly given

      • If you look for –ip then you may want to watch out

      • => batchspawner-singleuser

        • Arguments are injected for remote port selection but

        • Not parsing the command line

        • code:

          cmd_path = which(sys.argv[1])
sys.argv = sys.argv[1:] + ["--port={}".format(port)]
run_path(cmd_path, run_name="__main__")

        • Environment variable information carrying port info should defer to command line but needs test

      • … can we get port selection on the remote side (the singleuser executable)

        • Richard has a draft PR

          • 2 exclusive cases… need a scheme that works for both!

          • Can’t move everything over to the server because it doesn’t always have the info it needs (servers running in contairers know the bind port but not the connect port) - it can be handled but needs careful thought.

          • Could have a property of the spawner.

          • jupyterhub/jupyterhub#2727 (old, needs someone with either knoweledge or time to concentrate)

  • All: Standing project items:

    • Batchspawner check-in: Issues and PRs

      • Felix says he’ll try it out with Hub 2

        • Report to Discourse

    • Wrapspawner check-in: Issues and PRs

On this page
Edit on GitHub
Show Source